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This projects aims at testing different Bonification strategies for measuring to what extent 
the human auditory system can be used as primary interface channel for communicating, 
exploring, and interpreting physiological data, and its interactions in different contexts. 
Auditory display techniques and Sonic Interaction Design approaches (SID) are at the 
core of this project, in order to design a system capable of defining dynamic physiology- 
to-sound mappings, adapting to different purposes. In a musical context, we hypothesise 
that such approach will achieve better results than pre-define, ad hoc mappings in terms 
of perception, control and expressiveness. 
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Chapter 1 


Introduction 


1.1 Summary 

Research efforts in the field of cognitive neuroscience and physiological computing, make 
possible nowadays to extract meaningful and accurate mental state information from 
EEG brain activity[l][2]. Particularly, focusing on emotion psychological framework 
it is known that affective experiences are well characterised by two main dimensions: 
arousal and valence [3] [4], The dimension of valence ranges from highly positive to 
highly negative, whereas the dimension of arousal ranges from calming or soothing to 
exciting or agitating. This EEG data can be expressed via visual or sonic displays, as in 
the case of neurofeedback training paradigms[5]. In the musical domain, real time EEG 
data has also been used since 1965, when pioneers such as Alvin Lucier introduced it to 
the artistic domain, using audification of alpha waves for music composition [6]. 
Nevertheless, within the musical domain, the expressive capacities of human physiology 
as part of an interactive music system has never been explored in depth. In particular, 
the use of EEG has been rather specific in this domain, and the state of the art in 
this regard shows important limitations in terms of meaningful musical expression and 
control (low band-with, noisy signals, intrusive equipment, etc.) [7]. Artists such as Atau 
Tanaka and researchers from EAVI have pioneered this research line, but rather focusing 
in other types of bio-signals (EMG) [8] [9]. 

Coinciding the current state of the art on physiological computing and computer music, 
we aim at exploring the potential of EEG for expressive music control in a music per¬ 
formance scenario. 

In order to achieve this goal, we propose to estimate valence and arousal responses 
from players EEG activity in real time, and estimate a 2-dimension value within the 
valence-arousal space. Afterwards, this data will be mapped to some Reactable filters 
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and effects parameters via a bi-linear interpolation algorithm, i.e. associate emotive 
information from the performer to appropriate music control parameters. The direction 
of the mapping will be created ad hoc via associating 4 coordinates from valence-arousal 
state phase to 4 extreme 2 dimensional filters parameter values. 

In contemplation of system usability evaluation, we will also implement a gain controller 
object that will allow players to calibrate till what extent the EEG-driven mapping in¬ 
fluences the control of filters and effects. An experiment with Reactable expert users will 
be carried out in order to see if introduction of EEG-driven music control parameters 
will enhance music expressiveness and implicit control, showed by (i) players preference 
over conventional gestural control and (ii) a continuous use of the gain. 


1.2 Problem Statement 

The use of bio-signals and the works integrating together BCI and sound synthesis can 
be divided in two different approaches. The first one is the use of physiological data 
converted directly into sound in the so called process of sonification[10]. In the second 
approach, the musician tries to build a musical system that uses the physiological data 
for the control of the sound production[ll]. The project that we are presenting uses this 
second approach. 

Reviewing the state of the art, we can distinguish two different problematic points that 
have to be taken into consideration when using BCI in a musical domain. First one, 
regards the EEG signal acquisition. 

Non-invasive BCI approaches, such as Emotiv[12] and Neurosky[13] are becoming com¬ 
mon in these domain since they make use of low cost devices. Nevertheless, such devices 
are hard-coded and it is complicated to have access to its signal processing algorithms. 
Even more, they emit lower quantity signal than other higher-grade devices. Due to 
its wireless connections they can be more suited for a music performance scenario, yet 
possible user movements are more probable to process artifact signals and send them 
into the system[14]. Summing all up, we have decided to use a medical based EEG am¬ 
plifier device named Mitsar[15] and instead of using pre-coded EEG feature extracted 
signal we have build a Matlab based classifier for our emotional feature analysis, based 
on current research on EEG [2] [16]. 

The second problem arises, in the closing loop of the system, within the musical do¬ 
main. In order to get an expressive musical output from the performance we need, 
apart from having reliable EEG signalling, non-trivial EEG to sound mappings. Best 
mappings may be achieved when there is a perceptual connection between performer 
actions and their effect on the musical output [14] [17]. It has also been proven that using 
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self-customization mappings by the end users improves performance in a bio-feedback 
paradigm[18]. Therefore, we have implemented a brain mapping controller that will 
allow mapping preferences and adaptation by the performers. Moreover, we have based 
the mapping directions according to the musical into emotions research literature [19] [20]. 
In addition, most of the research in musical instruments that make use of physiological 
signals lack from user evaluation studies on system music expressiveness. That makes 
difficult to test, analyse and contribute further on mapping design strategies and overall 
system properties. Therefore, we will evaluate our system musical expressiveness within 
a methodological framework [21]. 


1.3 Architecture 


Our biological-controlled instrument will be mainly composed by three parts, as we can 
see in the figure below: 

• EEG Acquisition 

• Raw Data Processing 

• Reactable Control 


MITSAR EEG Studio Acquisition 



I 


Val/Aro OSC 


Pure Data 


P 

Pd 


Figure 1.1: Systems Diagram 


The first part of the system is composed by the EEG acquisition module. Mitsar EEG 
signal from the cap is amplified through the box and its raw signal is send to Matlab 
via a USB cable through EEGStudio software. This software that is used as a control 
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panel where you can set your cap montage, preprocesses raw data and filters it. 

Later on, in the second part of the system, Matlab receives that data and the scripts 
calculates valance and arousal indexes after a calibration stage. Then, those V/A signals 
ranging from 1-9 are directly send via OSC with an Ethernet cable. 

In the last stage, Pure Data receives OSC messages and uses them to control Reactable 
synth engine. Reactable objects, such as filters and effects are directly controlled with 
these OSC messages via a brain gain object. That brain gain object controls the final 
output of the system by interpolating tactile user parameters and valance-arousal ones 
received from Mitsar device. 

The system generates log files in all of the 3 stages: EEGStudio Record, Matlab raw 
data and valance-arousal data, and PD logs from brain gain object that are saved in a 
text file. 



Chapter 2 


Use of Physiology in Music 


Neural oscillations or brainwaves are a form of bioelectricity, or electrical phenomena in 
animals or plants. A neural oscillation is rhythmic or repetitive neural activity in the 
central nervous system. Neural tissue can generate oscillatory activity in different ways, 
driven either by mechanisms within individual neurons or by interactions between neu¬ 
rons. In individual neurons domain, oscillations can appear as oscillations in membrane 
potential or as rhythmic patterns of action potentials, which then produce oscillatory 
activation of post-synaptic neurons. At the level of neural ensembles, synchronised ac¬ 
tivity of large numbers of neurons can give rise to macroscopic oscillations, which can 
be observed in the electroencephalogram (EEG). 

EEG is described in terms of rhythmic activity and transients. The rhythmic activity is 
divided into bands by frequency. To some point, these frequency bands are a matter of 
nomenclature (i.e., any rhythmic activity between 8-12 Hz can be described as ’’alpha”), 
but these designations arose because rhythmic activity within a certain frequency range 
was noted to have a certain distribution over the scalp or a certain biological significance. 



Figure 2.1: EEG Raw signal of one second 


Human brainwaves were first measured in 1924 by Hans Berger. He termed these elec¬ 
trical measurements the electroencephalogram (EEG), which means literally ‘brain elec¬ 
tricity writing’. Berger first published his brainwave results in 1929 as “Uber das Elek- 
trenkephalogramm des Menschen” [22]. The English translation did not appear until 
1969. His results were verified by Adrian and Matthews in 1934 who also attempted to 
listen to the brainwave signals via an amplified speaker [23]. 
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Figure 2.2: Hans Berger, first EEG plots 

This was the first attempt to sonify human brainwaves for auditory display. The first 
instance of the intentional use of brainwaves to generate music did not occur until 
1965, when Alvin Lucier[6], who had begun working with physicist Edmond Dewan, 
composed a piece of music using brainwaves as the sole generative source. Music for 
Solo Performer was presented, with encouragement from John Cage, at the Rose Art 
Museum of Brandeis University in 1965. 



Figure 2.3: Music for Solo Performance, Alvin Lucier 


In the late 1960s, Richard Teitelbaum was a member of the innovative Rome-based live 
electronic music group Musica Elettronica Viva (MEV). In performances of Spacecraft 
(1967) he used various biological signals including brain (EEG) and cardiac (EGG) 
signals as control sources for electronic synthesisers. Over the next few years, Teitelbaum 
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continued to use EEG and other biological signals in his compositions and experiments 
as triggers for the nascent Moog electronic synthesiser. 



Figure 2.4: Musica Elettronica Viva (MEV) 

Then, in the late 1960s, another composer, David Rosenboom, began to use EEG sig¬ 
nals to generate music. In 1970-71 Rosenboom composed and performed Ecology of the 
Skin, in which ten live EEG performer-participants interactively generated immersive 
sonic/visual environments using custom-made electronic circuits. Around the same time, 
Rosenboom founded the Laboratory of Experimental Aesthetics at York University in 
Toronto, which encouraged pioneering collaborations between scientists and artists. For 
the better part of the 1970s, the laboratory undertook experimentation and research 
into the artistic possibilities of brainwaves and other biological signals in cybernetic 
biofeedback artistic systems. Many artists and musicians visited and worked at the 
facility during this time including John Cage, David Behrman, LaMonte Young, and 
Marian Zazeela. Some of the results of the work at this lab were published in the book 
“Biofeedback and the Arts” [24], A more recent monograph by Rosenboom, “Extended 
Musical Interface with the Human Nervous System” [25], remains the definitive theo¬ 
retical aesthetic document in this area. 

In France, scientist Roger Lafosse was doing research into brainwave systems and pro¬ 
posed, along with musique concrete pioneer Pierre Henry, a sophisticated live perfor¬ 
mance system known as Corticalart (art from the cerebral cortex). In a series of free 
performances done in 1971, along with generated electronic sounds, one saw a television 
image of Henry in dark sunglasses with electrodes hanging from his head, projected so 
that the content of his brainwaves changed the colour of the image according to his 
brainwave patterns. 
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Figure 2.5: Rosenboom while performing 



Figure 2.6: Vinil Cover of Corticalart 


Starting in the early 1970s, Jacques Vidal, a computer science researcher at UCLA, 
simultaneously began working to develop the first direct brain-computer interface (BCI) 
system using a IBM mainframe computer and other custom data acquisition equipment. 
In 1973, he published “Toward Direct Brain-Computer Communication” [26] based on 
this work. In 1990 Jonathan Wolpaw et al [27] at Albany developed a system to allow a 
user to exercise rudimentary control over a computer cursor via the alpha band of their 
EEG spectrum. Around the same time, Christoph Guger and Gert Pfurtscheller also 
began researching and developing BCI systems along similar lines in Graz, Austria 


In 2002, the principal BCI researchers in Albany and Graz published a comprehensive 
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Figure 2.7: Atau Tanaka in Biomuse 


survey of the state of the art in BCI research, “Brain-computer interfaces for commu¬ 
nication and control”. Then, in 2004, an issue dedicated to the broad sweep of current 
BCI research was published in IEEE Biomedical Transactions [28]. 

More recently, Brouse, Arsland and others [7] use EEG and EMG to control sound syn¬ 
thesis algorithms in order to build biologically driven musical instruments. A real time 
music synthesis environment and algorithms were developed to map these signals into 
sound. Finally, a ” bio-orchestra”, with two new digital musical instruments controlled 
by the EEGs and EMGs of two bio-musicians demonstrated this concept with a live 
concert on stage. 



Figure 2.8: BioOrchestra from Arsland et al. 




Chapter 3 


System Description 


In this chapter we will pursue a formal system description of our system. System Archi¬ 
tecture is mainly divided in 2 modules, as we can see in figure below 3.1, EEG Module 
and Reactable Module. Firstly, we will explain how in the EEG Module we process raw 
data information and we categorise it into Valence and Arousal dimensions, and later 
on we will explain Reactable Module, where we process those Valence and Arousal in 
order to control music parameters. 


EEG MODULE REACTABLE MODULE 


- SIGNAL ACQUISITION 


- INTERPOLATION 

- PREPROCESSING 


- MAPPING 

-CLASSIFICATION 


-CONTROL 


Figure 3.1: System Architecture Overview 


3.1 EEG MODULE: EEG Processing into Valance-Arousal 
Dimensions 

Figure above shows us a diagram that describes the EEG Module structure that we 
will discribe in this section. In that sense, firstly, we will give some context and foun¬ 
dations on EEG research, later on we will explain some of the research problems that 
arise working with affective states and physiological measures. Finally we will describe 
our EEG system architecture that will be divided into 3 main parts: signal acquisition, 
prepossessing and classification. 
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Figure 3.2: EEG Module Description 


3.1.1 Context 

Interactive music refers to a composition or improvisation in which software interprets 
live performance to produce music, which is generated or modified by computers. While 
arguably, not all the currently existing prototypes of devices for interactive music may 
present the same level of achievement or coherence, there are important reasons, perhaps 
often more intuited than stated, that turn live music performance and human computer 
interaction (HCI) in general, and musical instruments and tabletop tangible interfaces in 
particular, into promising and exiting fields of multidisciplinary research and experimen¬ 
tation. The reacTable, an ambitious tabletop musical instrument, is a good example of 
an instrument where musicians interact with computers, and it is a tool that facilitates 
the synergy between music and HCI. It is in fact, the design and use of computer-based 
tools for music composition and performance, which aims the development of diverse 
HCI’s like the one proposed in this thesis. 

The reacTable, it is played through specific objects that have different properties, such as 
generating raw sounds, playing samples, or manipulating qualities of the sounds played, 
with filters, adapting the sounds to specific keys, etc. The objects are handled in a 
very intuitive way in order to linearly change the settings of the computer generating 
the music. From the musical side, several computer music researchers have studied the 
control of sound in musical instruments as well as aspects of the communication between 
players and their instruments. A step further in this kind of communication, that may 
enhance the performance, would be not only “telling” the table directly what to do, but 
also including non-linear features, which aren’t in the absolute control of the musician. 
Concerning this idea, the reacTable is a multimodal music instrument where the incor¬ 
poration of new modalities such as based on biofeedback seems not only suitable but 
also exciting and useful from a musical point of view. This thesis focuses on the devel¬ 
opment of a new tool for music performance with reacTable, which will allow musicians 
to incorporate and map their emotional states of valence and arousal into the music. 
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3.1.2 Research Problem 

Understanding how information is encoded in the brain is a problem that raises more 
questions than answers. Through brain imaging techniques such as electroencephalog¬ 
raphy now it is possible to detect which areas in the cortex are active when a subject is 
performing a given task, such as lifting their hand,etc. For an experienced observer, it is 
not difficult to infer the cortical correlate of the assigned task by looking into the EEG 
activation patterns for each channel of the EGG device. But while the activation for 
motor tasks is fairly standard, there are other conditions, which cannot be deduced with 
such ease, as deviation between subjects is greater. Those are, among others, the neural 
correlate of users’ affective and emotional states, and the measure of their intensity in 
real time. 

From a technical scope, affective Computing deals with processes that relates to, arises 
from, or deliberately influences emotion or other affective phenomena. The analysis and 
recognition of any kind of information given from any way of human emotional expres¬ 
sion is a very profitable field that finds in the user’s advertisement customisation its 
maximum exponent, reason why it is very diverse and advanced the techniques of this 
sort. Besides, the development of artificial intelligent algorithms that make machines 
“feel” even the ambiguity of human irony in a text, would be useful in terms of deciding 
whether a brain “feels” bored or just relaxed, if the task would consist just in under¬ 
standing the cloud of features from EEG (or the implementation of a comprehension 
algorithm for relating the states of arousal/valence). Nevertheless it is not like that, 
again. Precisely because computing of this sort is popular, many projects are being 
developed to statistically analyse which parts of the brain are involved in specific emo¬ 
tional responses; there are many tools, theories and uncertainty, therefore it is often 
overwhelming to decide where to start testing. 

3.1.3 Proposed Solution 

Given the nature of the problem, the classification of the emotional states of Valence 
and Arousal of a musician playing the reac.Table the solution proposed for the analysis 
of those emotional states will lie in the implementation of a machine learning algorithm, 
trained to give a highly accurate value for the arousal state (on a scale from 1 to 9 on 
the abscissa) and valence mood (on a scale from 1 to 9 on the ordinate). The training of 
the preliminary filter will follow the method described in the paper “DEAP: A Database 
for Emotion Analysis Using Physiological Signals” from Sander Koelstra et ah, in which 
electrodes placed according to the international 10-20 system are studied, which are 
most relevant for analysing the emotional states from the rhythmic activity of theta, 
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alpha, beta and gamma bands - those most studied in association with arousal and 
valence. This correlation between cortical activity and emotion was possible due the 
exposure of 32 subjects to video clips, while music was rated in Last.fin in terms of level 
of valence, level of arousal, dominance and liking. That rating was recorded from the 
participants alongside EEG. All the data from both rating and the EEG database will 
be used in this thesis, as well as the statistical tools. After reproducing the results in the 
aforementioned paper, the validated tools will be used with new subjects, using their 
own ratings. So far, the machine learning tools proposed for classification will be based 
on, (in the first approach): 

• Ordinal regression. 

• Logistic regression. 

• Pattern recognition. 

• Neural Networks. 

• Support Vector Machines. 

3.1.4 Methods 

All the algorithms will be implemented in Matlab, since all the code resources needed 
have a toolbox in this language. Following the paper above, the procedure will be: 

For the features extraction: 

• Filter the signal in the band 3-47 Hz with Welch’s method. 

• Downsampling of the signal from 512 to 128 Hz. 

• Eye blinks removal with a blind source separation technique in the EEGlab toolbox. 

• Subtraction of the baseline power from signal. 

• Filter the rhythmic bands theta (3-7) Hz, alpha (8-13) Hz, beta (14-29) Hz, and 
gamma (30-47) Hz. 

• Correlation between power of rhythmic activity frequency bands and ratings due 
the Spearman’s correlation coefficients. 

• P-values obtained from the coefficients that will correlate direction (valence/arousal 
1 to 9 rating), frequency band and electrode will merge into one p-value via Fis¬ 
cher’s method. 
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For the classification: 

• The Fischer’s linear discriminant J selects relevant features. 

• A naive Bayes statistical classifier for independence assumptions, that has been 
proven to work well with unbalanced classes, will discriminate which features be¬ 
long to which classes (1 to 9, valence/arousal). 

• In the article the F 1-core for binary classifiers is employed in order to give relevant 
information of classes and the balance. So, for each participant, there will be a 
score that will evaluate their performance of emotion classification. 

3.1.5 Overall EEG Module Description 

In this part, we present an overall technical resume of EEG Module functioning. 

1. Signal Adquisition: Mitsar Amplifier 

• 10-20 International Electrode Placement System. 

• F3 and F4 channels. 

• Al, A2 lobe channels as reference. 

• CZ ground electrode. 

2. DSP: EEGStudio Software 

• Filtering (Low, High, Noch). 

• Downsampling to 128 Hz. 

• Rhytmic Filter (alpha, beta). 

3. Valence/Arousal Classification: Matlab 

• Power Spectra in frequency domain (FFT). 

• Valance: beta f3 + beta f4 / alpha f3 + alpha f4 

• Arousal: alpha f4 / beta f4 + alpha f3 / beta f3 

3.2 REACTABLE MODULE: From Affective States to Mu¬ 
sical Parameters 

In this section, we will explain our two main approaches used in order to build up a 
system capable of transposing affective states into a musical output. First system, will 
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be based on a global controller for some of the Reactable objects, while the second one 
will be based on a low-frequency oscillator that will act in a local manner. 

3.2.1 Theoretical Framework 

Music has been widely accepted as one of the languages of emotions. The possibility to 
select appropriate affective music can be helpful to adapt music to our emotional inter¬ 
est. Nevertheless, only recently scientists have tried to quantify and explain how music 
influences our emotional states. According to Scherer (1984), emotions may be con¬ 
ceived as consisting of various components: cognitive appraisal, physiological activation, 
motor expression, behaviour intentions, and subjective feeling. Emotional states can be 
described as particular configurations of these components.For a long time, cognitive 
sciences have been studying the foundations of emotions. More recently, computational 
models have also been proposed and are being applied in several domains (e.g. music, 
dance, and cinema). There are distinct approaches and techniques used to generate 
music with appropriate affective content. Starting from results of previous work (Schu¬ 
bert 1999), Livingstone and Brown [29] established relations between music features and 
emotions. Both emotions and a set of music emotion structural rules were represented 
in a 2-Dimensional Emotion Space with an octal form. 


Active 



Passive 

Figure 3.3: Primary V/A music categories 


For our approach into musical emotions we will follow that approach that combines 
categorical and dimensional approaches to emotion expression to illustrate that the code 
allows performers to communicate both graded signals (level of activity) and categorical 
signals (happiness). Each emotional expression is placed at an approximate point in a 
two dimensional emotion space constituted by valance and activity level [3]. 
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3.2.2 Reactable BCI System - Global Control 


Readable main parameter of filters and effects, accessed by rotation of the object, and 
second parameter, controlled with finger movement around the object, will be controlled 
by EEG V/A features through a bi-linear interpolation. 

In mathematics, bilinear interpolation is an extension of linear interpolation for inter¬ 
polating functions of two variables (e.g., x and y) on a regular 2D grid. 


Yz 




y 



y. 




X-I X 


*2 


Figure 3.4: 2D grid for linear interpolation 


Mathematically, the desired interpolation point is calculated as this: 
/(*.»)« ) (*2 - x){V2 -y) + 

»i)+ 

(x2-S-yi ) (^~ ^i)(y-yi) 

(xa-xtto-y!) (/(<?n)(x 2 - x){y 2 - y) + 

f{Q 2 i)(x - X- L ){y 2 -y) + 

f(Qi 2 )(x2-x){y-yi) + 

f(Q22){x~xi){y-yi) ) 


Our system implemented in PD, figures above, sends Arousal and Valance values re¬ 
ceived from Matlab to two distinct interpolators that control in a loop both musical 
parameters from Reactable objects. Brain gain object controls the overall values, in a 
second interpolation stage, send to object parameter by matching tangible user param¬ 
eters and the ones received from BCI. 
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expr (1.0 / 4.0) * (($f5 - $f6 - $f4 + $£3) * Ifl + $f5 + 
$f6 - $£4 - $f3) * Sf2 + (1.0 / 4.0) * (($f5 - $£6 + $f4 
$£3) * $fl + $£6 + $£5 + $f4 + $f3) 

E 


Figure 3.5: PD interpoler 



Figure 3.6: Overall PD interpoler with Brain Gain object 



Figure 3.7: Interpolation Schema 
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Above we can see an example diagram of the brain object controlling a filter, and also 
the same architecture implemented in PD. When the brain gain object is at maximum 
value the object parameters are totally controlled by EEG features. In the case when 
brain object is at zero, the object parameters can only be controlled by tactile move¬ 
ments from the user. In other cases, the final parameter values are the result of an 
interpolation between tactile settings and BCI features. 




Figure 3.8: Brain object controlling a Filter 




Figure 3.9: PD filter architecture 

























































































System Description 


19 


3.2.2.1 Mapping on Readable Objects 

The objects that will be controlled are a filter, a delay, a distorter and a modulator. 
The brain gain object will act globally, effecting all the objects mentioned above placed 
in the table during performance. The mappings are made accordingly to music timbre 
characteristics. The four extreme values for the Reactable system in the interpolation 
mapping are indicated below: 


• Filter: 

— Valance-Arousal 
— Valance-Arousal 
— Valance-Arousal 
— Valance-Arousal 

• Delay: 

— Valance-Arousal 
— Valance-Arousal 
— Valance-Arousal 
— Valance-Arousal 

• Distortion: 

— Valance-Arousal 
— Valance-Arousal 
— Valance-Arousal 
— Valance-Arousal 

• Modulator/Chorus: 

— Valance-Arousal 
— Valance-Arousal 
— Valance-Arousal 
— Valance-Arousal 


(-1,-1) to CuttOff-Resonance (10,0) 
(-1,1) to CuttOff-Resonance (10,1) 
(1,-1) to CuttOff-Resonance (127,0) 

(1,1) to CuttOff-Resonance (127,1) 


(-1,-1) to Delay Time-Feedback Amount (0,0) 
(-1,1) to Delay Time-Feedback Amount (0,1) 
(1,-1) to Delay Time-Feedback Amount (8,0) 

(1,1) to Delay Time-Feedback Amount (8,1) 


(-1,-1) to Main Parameter-Dry/Wet (0,0) 
(-1,1) to Main Parameter-Dry/Wet (0,1) 
(1,-1) Main Parameter-Dry/Wet (4,0) 

(1,1) to Main Parameter-Dry/Wet (4,1) 


(-1,-1) to Main Parameter-Dry/Wet (0,0) 
(-1,1) to Main Parameter-Dry/Wet (0,1) 
(1,-1) Main Parameter-Dry/Wet (10,0) 

(1,1) to Main Parameter-Dry/Wet (10,1) 


Other effects such as granulator and ring modulator will be implemented for the Re- 
actable BCI system 
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Figure 3.10: Readable Effects and Filter 


3.2.3 Reactable BCI System - Local Control 

This local system, like global one, also allows players to calibrate till what extent the 
EEG-driven mapping influences the objects. Nevertheless, the way this influence is 
adjusted is achieved via an amplitude parameter, i.e. the distance of the lfo brain object 
to the controlled Reactable object. In that sense, when the local brain object is near the 
Reactable object the amount of BCI signal will be maximum. On the contrary, when 
the distance of the brain lfo to the Reactable object is augmented, the influence of BCI 
signal decreases. 

Local system acts reassembling an lfo. In that sense, brain local objects controls a 
delta increase or decrease over central values set-up by user preferences. The amount 
of delta increase-decrease as we mentioned above, is controlled by the distance of the 
brain local gain to the object. In addition, finger parameter of the local controller can 
be manipulated in order to change sample and hold parameter values. This quantisation 
feature allows to send range of values in a discrete or in a continuum manner. Moreover, 
lfo brain object allows BCI signals to be triggered according to overall Reactable tempo. 
In that sense all BCI changes are triggered taking into account musical dynamics in 
performance. 

Figure above, shows PD patch for the local brain object. We can observed that BCI 
signal output from pd VA interpolation box is later processed with a line object with a 
time frame of Is and is later input to a an audio signal object. 
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Chapter 4 


Experimental Design 


In this chapter, we will be presenting the overall experimental design carried out in order 
to test 2 different hypothesis: 

• HI: There will be an augment in music expressiveness incorporating physiology- 
driven music control into Reactable. 

• H2: There will be a neuro-feedback effect when using physiology-driven music 
control into reactable. 


4.1 Experiment Description 

Our main research hypothesis (HI) states that it is possible to integrate EEG brain 
signal using affective valence arousal descriptors in order to control music in expressive 
ways. Particularly, we will focus our musical framework within the Reactable table- 
top tangible interface, using some of its effects and filters parameters augmented via a 
“brain” object. That object will be implemented in two different ways and will manage 
the amount of mapping that it is send to the filters and effects . In Condition 1, we 
will be using a Global Brain Object and for Condition 2, the Local Brain Object. In 
contemplation of the evaluation of our hypothesis we propose to develop two different 
experimental conditions for one target group of Reactable Expert Users. 


• Sample: 5 Reactable Expert Users. 

• Repeated Measures. 

• 2 Conditions: 
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— Cl - Global Control 
— C2 - Local Control 

• 2 Music Composition Tasks 

— Tl- Positive Valence (Happy) and High Arousal (Active). 

— T2- Negative Valence (Sad) and Low Arousal (Calm). 

• Duration: 1 h. 

• Counterbalanced conditions and tasks. 

• Previous system training session before 1st task. 

Two subjective music composition tasks will be performed by five different users in 
two different sessions in each condition. They will have the availability of the full set 
of generators, effects and filters, controllers and global controllers from Reactable plus 
the “brain” controller object. Whereas users will have the restriction to perform with 
“brain” object on the table, they will have the possibility to move its gain from 0 to 
1, meaning null variation from user defined parameters to full brain mapping control. 
That object functionality will be masked to users to prevent conditioning them. Once 
music performance is finished, a subjective and objective user experience evaluation 
method will be used. Firstly, an analysis of the observation of the users will be carried 
out together with an interview and a questionnaire. Secondly, we will analyse the gain 
of the log file from the “brain” object together with the user EEG recorded feedback. 
Last session, will serves us to control the differences between initial tasks and final tasks. 


4.2 Experiment Protocol 

The experiment lasts about 1 hour according the following protocol: 

1. Information and consent form: The participant is explained each stage of the 
experiment, and the relation between brain activity and Reactable sound mapping; 
the consent form is signed (3 minutes). 

2. Sensor placement and baseline state recording: The participant is sat in front 
Reactable, and the Mitsar is mounted. Impedance check j 5 kOm. Blinking, 
closing eyes - checking that recording is ok. The baseline EEG activity is recorded 
for 2 minutes - eyes open, no motion. (10 minutes). File record. 
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3. Exploratory Phase: the experiment coordinator explains how the system works. 

4. Experiment coordinator performs a live demonstration of how the system works 

5. Once the explanation is done, the participant has 5 minutes to use the system, 
check the available sounds and make any question to the supervisor. The supervisor 
will assist the participant during the exploration phase (8 minutes). 

6. (SAMI) Pre-test emotional state self-assessment: The participant fills two SAM 
scales (Subjective measures of emotional valence and arousal collected in paper 
through a 9-point Self-Assessment Manikin (Bradley and Lang, 1994)) 

7. Relaxation induction and calibration: The participant is asked to sit down in a 
chair and try to relax while listens a sound of waves during 2 minutes. EEG 
activity is recorded (2 minutes). File record. 

8. (SAM2) Pre-test emotional state self-assessment (2 minutes). 

9. (Impedance check before!) Task 1: The participant is asked to perform a musical 
task, either a exciting-happy or a relaxing-sad composition. The order of tasks 
will be randomised (10 minutes). File record. 

10. (SAM3) Post-task 1 emotional state self-assessment (2 minutes) 

11. Music Meaningfulness questionnaire (3 minutes) 

12. Sensor removal (3 minutes) 

13. Music Meaningfulness questionnaire against Reactable (3 minutes) 

14. Interview (5 minutes) (only in the second session). 

15. Debriefing 

4.3 Experiment Explanation 

In this section we can find the experiment description for Reactable expert user partic¬ 
ipants: 

In this experiment we will assess the role of physiology-based, interaction in music per¬ 
formance. We use EEG activity to estimate users ’ affective responses that later will be 
sent to a tabletop interface (Reactable) for controlling music parameters. For measuring 
EEG activity we use wet electrodes placed in the scalp. The experiment is composed of 
two sessions of 1 hour, to take place in different days. During the experiment you will be 
asked to perform two musical tasks using the Reactable. During the experiment we will 
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collect subjective measures regarding demography, music knowledge, music expressive¬ 
ness, musicality, control and affective states. In the same manner, we will also collect 
performance and physiological data, together with video recordings. All the information 
will remain anonymous and will be used exclusively for analysis purposes. At the end 
of the second session we will also carry on interviews to gather feedback through open 
ended questions. 


4.4 Music Meaningfulness Questionnaire 

Here we show the likert scale questionnaires used to test music meaningfulness. 

The System’s properties are measured according to 3 variables: 

• Mapping richness. Statement: I have found the control mapping rich and inter¬ 
esting” . 

• Synthesis richness. Statement: I have found the sound synthesis rich and interest¬ 
ing”. 

• Potential. Statement: The system shows great potential as a DMI”. 

Performance’s aspects, on the other hand, were assessed through the following variables: 

• Musicality. Statement: I have found the performance musical”. 

• Expressiveness. Statement: I have found the performance expressive”. 

• Virtuosity. Statement: The performers were able to control the instrument as real 
virtuosi”. 


4.5 Experimental Data: Measures and Statistical Analysis 

From EEG module, we will get raw data from Matlab for the baseline. That recording 
will be compared with two tasks recordings, in order to see if the whole EEG acquisition 
systems is working under correct circumstances. Recordings from V/A indexes from 
Matlab will then be compared to SAM scales obtain by user ratings in each protocol 
steps. 

Text files containing the use of brain gain button will be analysed together with music 
meaningfulness questionnaires and subjective measures extracted from video recordings. 
That will compose the overall data in Reactable module. 
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Finally, a comparison between EEG module measures and Reactable module measures 
will be performed to get finally results that will reinforce or not our research hypothesis. 
Next, we present a list of data measures that will be acquired during all experimental 
procedures. 


• Main Research Hypothesis: ” It is possible to integrate EEG brain signal using 
affective valence arousal descriptors in order to control music in expressive ways” 

• Main 0-Hypothesis: ” It is not possible to integrate EEG brain signal using affec¬ 
tive valence arousal descriptors in order to control music in expressive ways” 

• Secondary Research Hypothesis: ” There will be a neurofeedback effect when using 
physiology-driven music control into reactable. ” 

• Secondary 0-Hypothesis: ” There will not be a neurofeedback effect when using 
physiology-driven music control into reactable. ” 

Measures: 


1. Affective Responses Part 

• Raw EEG data 

• V/A indexes 

• SAM scale questionnaires 

2. Expressiveness Part 

• Music Meaningfulness Questionnaires 

• Brain Gain log hie 

• Open Rrterviews 

Statistical Analysis: 

• Statistical Assumptions (Statistical Independence, Distribution). 

• Test Statistic T. 

• Distribution of 0-Hypothesis. 

• a Significance Level. 


Pearson Correlation Analysis. 



Chapter 5 


Results 


This chapter shows the analysis of the results obtained after carrying a total of 10 exper¬ 
iments with 5 subjects under 2 conditions. For each measure, affective and expressive, 
we present results both from condition 1 (global) and condition 2 (local). 


5.1 Affective Responses Results 

5.1.1 Condition 1 

In the figure above 5.1 we can see the mean valence and arousal values, (ranging from 
1 to 9) with their standard deviation error bars, of 5 subjects under 3 tasks. Task 0 
represent the baseline recording while hearing during 2 minutes a relaxing sound of sea 
waves. Task 1 represents first high valence and high arousal music composition. Task 2 
represents low valence and low arousal music composition. 
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Figure 5.1: Mean Valence and Arousal Values from EEG Classification - Cond.l 


Taking a look in to the graphic 5.1, we can see a slight variation between tasks in the 
expected direction. Nevertheless, it is not significant so far (valence p = 0, 28) (arousal 

p = 0,612). 



Figure 5.2: Mean Valence and Arousal Values from SAM scales - Cond.l 


From SAM valence and arousal scales 5.2, we can see close to significant variations 
between task 1 and task 2 (valence p = 0,071) (arousal p = 0,083). 
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ANOVA 



Suma de 
cuadrados 

gi 

Media 

cuadratica 

F 

Sig. 

Valence 

Entre grupos 

.142 

2 

,071 

1,417 

,280 


Dentro de 
grupos 

.600 

12 

,050 




Total 

,741 

14 




Arousal 

Entre grupos 

,020 

2 

,010 

.512 

,612 


Dentro de 
grupos 

,238 

12 

,020 




Total 

,259 

14 




SAMv 

Entre grupos 

3,333 

2 

1,667 

3,333 

,071 


Dentro de 
grupos 

6,000 

12 

,500 




Total 

9,333 

14 




SAMa 

Entre grupos 

5,733 

2 

2,867 

,915 

,083 


Dentro de 
grupos 

37,600 

12 

3,133 




Total 

43,333 

14 





Figure 5.3: ANOVA analysis between tasks of EEG and SAM indexes - Cond.l 


5.1.2 Condition 2 

In figures below 5.4, 5.5, we show the affective responses measures for sample group of 5 
subjects within condition 2. In this case, Task 0 also represents the baseline recording. 
Yet, after counterbalancing between conditions, Task 2 represents high valence and high 
arousal music composition and Task 1 represents low valence and low arousal music 
composition. 



Figure 5.4: Mean Valence and Arousal Values from EEG Classification - Cond.2 
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Figure 5.5: Mean Valence and Arousal Values from SAM scales - Cond.2 


Tables 5.6, 5. 
measures and 


7 below shows descriptive statistics analysis both from EEG and SAM 
ANOVA analysis. 


Descriptivos 



N 

Media 

Desviacion 

estandar 

Error 

estandar 

95% del intervalo de 
confianza para la media 

Minimo 

Maximo 

Limite 

inferior 

Limite 

superior 

ValenceEEG .00 

5 

4,9320 

,15156 

.06778 

4,7438 

5,1202 

4.76 

5,13 

1,00 

5 

5.0440 

,18501 

,08274 

4,8143 

5,2737 

4.76 

5,27 

2,00 

5 

4,9960 

,07301 

,03265 

4,9054 

5,0866 

4,88 

5,08 

Total 

IS 

4,9907 

.14185 

,03663 

4.9121 

5,0692 

4.76 

5,27 

ArousalEEG .00 

5 

4,5140 

.10877 

.04864 

4.3789 

4.6491 

4.36 

4.61 

1,00 

5 

4,4960 

,11653 

,05212 

4,3513 

4,6407 

4,36 

4,67 

2,00 

5 

4,5140 

,08905 

,03982 

4,4034 

4,6246 

4,39 

4,59 

Total 

IS 

4,5080 

,09799 

,02530 

4,4537 

4,5623 

4,36 

4,67 

ValenceSAM .00 

5 

6.8000 

.44721 

.20000 

6,2447 

7,3553 

6.00 

7,00 

1,00 

5 

6,4000 

,89443 

,40000 

5,2894 

7,5106 

5,00 

7,00 

2,00 

5 

5,8000 

1,64317 

.73485 

3,7597 

7,8403 

4,00 

8,00 

Total 

IS 

6,3333 

1,11270 

.28730 

5,7171 

6,9495 

4,00 

8,00 

ArousalSAm .00 

5 

4.4000 

1.51658 

.67823 

2,5169 

6,2831 

3,00 

6.00 

1,00 

5 

3,4000 

,89443 

,40000 

2,2894 

4,5106 

3,00 

5,00 

2,00 

5 

6,0000 

1,87083 

,83666 

3,6771 

8,3229 

4,00 

8,00 

Total 

15 

4,6000 

1.76473 

.45565 

3,6227 

5,5773 

3,00 

8,00 


Figure 5.6: Descriptive Statistics of EEG and SAM scales - Cond.2 
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ANOVA 



Suma de 
cuadrados 

gi 

Media 

cuadratica 

F 

Sig. 

ValenceEEG 

Entre grupos 

.032 

2 

,016 

.757 

,490 


Dentro de 
grupos 

.250 

12 

.021 




Total 

,282 

14 




ArousalEEG 

Entre grupos 

,001 

2 

,001 

,049 

,953 


Dentro de 
grupos 

.133 

12 

,011 




Total 

.134 

14 




ValenceSAM 

Entre grupos 

2,533 

2 

1,267 

1,027 

,388 


Dentro de 
grupos 

14,800 

12 

1,233 




Total 

17,333 

14 




ArousalSAm 

Entre grupos 

17,200 

2 

8,600 

3,909 

,049 


Dentro de 
grupos 

26,400 

12 

2,200 




Total 

43,600 

14 





Figure 5.7: ANOVA analysis between tasks of EEG and SAM indexes - Cond.2 


By inspecting the data presented above, we can observe similar trends between EEG 
values and those from the SAM scales. Both arousal values show an increase between 
tasks in the expected direction, with a significance variation for the arousal SAM scale 
rating (p = 0.049). Tough, valence does not move into the expected direction, i.e. an 
increase in valence between task 1 that was low and task 2 that was high. In that sense, 
both valance affective responses measures decrease between tasks. Nevertheless, we do 
not find any significant differences for this last cases. 


5.2 Expressiveness Results 

5.2.1 Condition 1 

Within expressiveness evaluation part, we can see that the mean value of the Brain 
Gain slight decreases between task 1 (0,71) and task 2 (0,64), nonetheless, we find no 
significance in these results (p = 0, 553) 5.8. We can also see and increase valuation 
in System and Performance aspects that are close to significant 5.9. In system case we 
find a nearly to significant value (p = 0,067), yet in performance we see no significance 
(p = 0,471). Furthermore, we can observe a negative correlation between Brain Gain 
and System aspects, where we find a significant Pearson correlation ( rho = —0, 783, 
a = 0,007). 
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Task 


Figure 5.8: Mean Use of brain Gain and Likert Scale Ratings - Cond.l 


ANOVA 



Suma de 
cuadrados 

gi 

Media 

cuadratica 

F 

Sig. 

BrainGain 

Entre grupos 

,011 

1 

,011 

,384 

,553 


Dentro de 
grupos 

,227 

8 

,028 




Total 

,238 

9 




System 

Entre grupos 

,900 

1 

,900 

4,500 

,067 


Dentro de 
grupos 

1,600 

8 

,200 




Total 

2,500 

9 




Performance 

Entre grupos 

,400 

1 

.400 

,571 

.471 


Dentro de 
grupos 

S,600 

8 

,700 




Total 

6,000 

9 





Figure 5.9: ANOVA analysis between tasks of Brain Gain and Expressiveness Ques¬ 
tionnaires - Cond.l 


5.2.2 Condition 2 

For condition 2 5.10, within expressiveness evaluation part, we can see that the mean 
value of the Brain Gain slight decreases between task 1 (0,67) and task 2 (0,51), 5.11. 
Nonetheless, we find no significance in these results (p = 0, 31), 5.12. We can also see an 
decrease valuation in System and Performance aspects between task, nonetheless, those 
are not significant (p = 0,57)(p = 0,69), 5.12. Additionally, looking for correlations 
in all the measures, we can observe a positive correlation between system aspects and 
EEG arousal, where we observe a close to significant Pearson correlation ( rho = —0, 73, 
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a = 0,017), 5.13. 
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Figure 5.10: Mean Use of brain Gain and Likert Scale Ratings - Cond.2 


Descriptivos 



N 

Media 

Desviacion 

estandar 

Error 

estandar 

95% del intervalo de 
confianza para la media 

Mi'nimo 

Maximo 

Li mite 

inferior 

Limite 

superior 

System 

1,00 

5 

3,0000 

1,00000 

,44721 

1,7583 

4,2417 

2,00 

4.00 


2,00 

5 

2.6000 

1,14018 

,50990 

1,1843 

4,0157 

1,00 

4,00 


Total 

10 

2,8000 

1,03280 

,32660 

2,0612 

3,5388 

1.00 

4.00 

Performance 

1.00 

5 

3.2000 

,83666 

,37417 

2.1611 

4,2389 

2,00 

4.00 


2.00 

5 

3.0000 

,70711 

.31623 

2,1220 

3,8780 

2,00 

4.00 


Total 

10 

3.1000 

.73786 

,23333 

2,5722 

3,6278 

2,00 

4.00 

BrainGain 

1.00 

5 

.6860 

.18596 

.08316 

.4551 

.9169 

.44 

.88 


2,00 

5 

.5140 

.29838 

,13344 

,1435 

.8845 

.11 

,92 


Total 

10 

,6000 

,25131 

,07947 

,4202 

,7798 

,11 

,92 


Figure 5.11: Descriptive Statistics of brain Gain and Likert Scale Ratings - Cond.2 


ANOVA 



Suma de 
cuadrados 

gi 

Media 

cuadratica 

F 

Sig. 

System 

Entre grupos 

,400 

1 

,400 

,348 

,572 


Dentro de 
grupos 

9,200 

8 

1,150 




Total 

9,600 

9 




Performance 

Entre grupos 

.100 

i 

,100 

,167 

,694 


Dentro de 
grupos 

4,800 

8 

,600 




Total 

4,900 

9 




BrainCain 

Entre grupos 

,074 

i 

,074 

1,197 

,306 


Dentro de 
grupos 

,494 

8 

,062 




Total 

,568 

9 





Figure 5.12: ANOVA of brain Gain and Likert Scale Ratings - Cond.2 
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Correlaciones 


1 1 

ArousalEEG 

ArousalSAm 

ValenceEEG 

ValenceSAM 

System 

Performance 

BrainGain 

ArousalEEG 

Correlacion de 
Pearson 

1 

,222 

,367 

-.033 

,726* 

.473 

,061 


Sig. (bilateral) 


,426 

,178 

,908 

,017 

,168 

,867 


N 

IS 

15 

15 

15 

10 

10 

10 

ArousalSAm 

Correlacion de 
Pearson 

,222 

1 

-.016 

,182 

,318 

.129 

.474 


Sig. (bilateral) 

.426 


.955 

,516 

.371 

,723 

,166 


N 

15 

15 

15 

15 

10 

10 

10 

ValenceEEG 

Correlacion de 
Pearson 

,367 

-.016 

1 

-.450 

,099 

,117 

-.249 


Sig. (bilateral) 

,178 

,955 


,093 

,786 

,748 

,489 


N 

IS 

15 

15 

15 

10 

10 

10 

ValenceSAM 

Correlacion de 
Pearson 

-.033 

,182 

-.450 

1 

-.431 

-.345 

-.139 


Sig. (bilateral) 

,908 

,516 

,093 


.214 

,330 

,702 


N 

IS 

15 

15 

15 

10 

10 

10 

System 

Correlacion de 
Pearson 

,726* 

,318 

,099 

-.431 

1 

,467 

.325 


Sig. (bilateral) 

,017 

.371 

,786 

.214 


.174 

.359 


N 

10 

10 

10 

10 

10 

10 

10 

Performance 

Correlacion de 
Pearson 

.473 

,129 

,117 

-.345 

.467 

1 

-.048 


Sig. (bilateral) 

,168 

.723 

,748 

,330 

.174 


,895 


N 

10 

10 

10 

10 

10 

10 

10 

BrainGain 

Correlacion de 
Pearson 

,061 

,474 

-.249 

-.139 

.325 

-.048 

1 


Sig. (bilateral) 

,867 

,166 

,489 

,702 

.359 

,895 



N 

10 

10 

10 

10 

10 

10 

10 


Figure 5.13: Bivariate Pearson Correlation Analysis of Affective Responses and Ex¬ 
pressiveness Measures - Cond.2 


5.3 Open Questionnaires and Interviews Results 


Here we present other questionnaire results regarding different aspects other than system 
or performance ones. Table below shows Likert Scale ratings from 1 to 5, where 1 means 
’’strongly disagree” and 5 ’’completely agree”. First item number 1 stands for global 
control condition, whereas second item 2 stands for local control condition. 

• I loose control over music composition during performance: 

1. 3 

2. 3 

• I liked musical timbre variations induced by the system. 

1. 3 

2. 3 

• I enjoyed playing b-Reactable instrument. 

1. 4 

2. 3 

• I would like to play with b-Reactable instrument in the future. 
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2. 3 

• I think b-Reactable instrument fits well into a musical performance scenario. 

1 . 2 
2. 3 


Furthermore, we present some interesting points addressed by participants in an open 
interview after having performed last experiment. 


• Subject ID 1: 

— Global control has less control possibilities than local control, this one, is more 
natural and integrated in Readable framework and is less restrictive. 

— I used both systems just if they were inputting a random signal. I didn't 
understand if there was a relation between my affective responses and system 
musical direction. I couldn’t plan my performance based on BCI system. 

• Subject ID 2: 

— I would like to have a mixed controlled system both local and global. 

— I felt the need of more sudden changes in music output when using brain 
objects. 

— It would be nice to have more brain Ifo objects (local controllers) in order to 
apply them to different Readable objects. 

• Subject ID 3: 

— I didn’t feel how the mapping was controlling music parameters yet I think 
it was design in the right way. I use signals as if they were random instead 
coming from my affective states. 

— This system could have sense in a neurofeedback training paradigm but non 
in a performance scenario, equipment is also so intrusive. 

• Subject ID 4: 

— I didn’t feel my affective state reflected in music tasks. I felt it was more 
random than it was showing a tendency. Maybe in local control the effect is 
less perceptive than in global control. 

— In global control, music events were happening in an unexpected way and in 
local control the effects were smooth because there was only one object to be 
controlled. 
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— I would include a graphic mapping application in order to customise mapping. 
For instance, have a GUI where you can see your valence and arousal state 
and other GUI to change music parameters according to first GUI. 

• Subject ID 5: 

— I only could feel smooth changes during both conditions. In that sense, I 
would like to change other music parameters like tonality with my valence 
and arousal. 

— I would implement more brain Ifo’s for the local control just to change more 
than one object at time. 



Chapter 6 


Discussion 


In this last chapter we will present an overall discussion of results that we show in 
chapter 4. First, we will discuss results from both conditions and from affective responses 
and music expressiveness measures. Later on, we will discuss the important differences 
between both conditions taking into account also the open interviews. 


6.1 Global Brain Condition 

From affective measures, even there is no significance difference in EEG levels between 
tasks (valence p = 0, 28) (arousal p = 0,612), users affective responses change at a sub¬ 
jective level (valence p = 0,071) (arousal p = 0,083). Considering that both measures 
change in the same direction, we can say that there is an slightly moderate neuro¬ 
feedback effect using the system. 

From music expressiveness measures, we see that valuation on the system improves after 
tasks (close to significant, p = 0, 067) due to a decrease use of the Brain Gain (correlated, 
p = —0,783, er = 0,007). Nevertheless, there is no significance in the use of the Brain 
Gain and in performance aspects between tasks (p = 0,471). In that sense, we can not 
clearly affirm or deny that there has been an increase in music expressiveness using the 
system. 


6.2 Local Brain Condition 

From affective measures, there are no significance differences in EEG levels between 
tasks (valence p = 0,49) (arousal p = 0,95), and users affective responses change at a 
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subjective level only for the arousal (valence p = 0, 38) (arousal p = 0,049). Taking into 
account that only arousal measures change in the same direction, we can not say that 
there is significant neuro-feedback effect using local control system. 

Within expressiveness evaluation part, we can see that the mean value of the Brain Gain 
slight decreases between task 1 (0,67) and task 2 (0,51), yet, we find no significance in 
these results (p = 0,31). We can also see a decrease valuation in System and Performance 
aspects between tasks, though, those are not significant (p = 0,57)(p = 0,69). In 
addition, bi-variate analysis shows positive correlation between system aspects and EEG 
arousal (p = 0, 73,ct = 0,017). Summing all up, we can point out that (even there is 
no relevant significance) when the use of the brain gain is lower also the participants 
ratings on music expressiveness decrease. 


6.3 Comparison Between Condition Cl and C2 


There are no significant differences between Cl and C2 regarding both affective measures 
and music expressiveness ones (p > 0,05). Nevertheless, we see a consistency between 
task direction and SAM subjective evaluations in both conditions, and similarly, con¬ 
sistency between arousal EEG measures and task direction. That could mean that the 
arousal classifier is more reliable than the valence one and more effort has to be applied 
to this last one. 

In music expressiveness part, we see that system and performance aspects ratings are 
lower, both for Cl and C2, in high valence and high arousal tasks and are higher in 
low valence and low arousal tasks. This result may be explained by the fact that high 
valence and high arousal task need a major number of elements placed in the table. 
When this situation happens in a global control scenario it is easier to lose control over 
those objects. In a local control scenario for that task participants can loose the feeling 
of controlling the composition since they only have the possibility to change one object 
from the multiple ones placed in the table. 

In addition, brain gain mean use is always above 50% in both conditions, so participants 
don’t discard its function for composing their musical tasks. The fact that brain gain 
object mean use is always lower in task 2 than in task 1 can be explained by the occur¬ 
rence when participants get use to it. 

Summing all up, together with the open question interviews, we see that both systems 
have its pros an cons. From one hand, global control system is capable of augmenting 
the feeling that BCI signals are influencing in a wider range the overall music output. 
Tough, this can be diminishing performance control over composition. Local control 
system otherwise, has happened to be more customisable, and in this way it creates 
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a more accurate control when performing music tasks. Nevertheless, the music output 
effect it is not so noticeable since performer can only influence one object at time. 


6.4 Future Work Directions 

More experiments have to be carried out in order to assess the role of EEG physiology 
for the control of music systems in expressive ways. From one hand, this study has been 
rater focused on a particularly narrow group of Reactable expert users. Maybe, opening 
the system also to electronic music experts, and music experts in general, can lead us to 
wider results. 

A between 3 conditions design can be done for this new target group in which it will have 
to perform a musical oriented task. The first condition will be with normal Reactable 
system, the second condition will perform with Reactable with the Global Controller and 
third condition will be with Reactable with Local Controller . The same user evaluation 
method like in Reactable Experts group will be carried out together with an analysis if 
they have reach or not the fixed musical target. 

Moreover, an important point to be addressed is the use of robust and reliable affective 
responses classifiers, and specially for the valence one. This research domain into affec¬ 
tive computing is quite new, and there is not a lot of literature a part from Koelstra et 
al. [2] . Other classification techniques have to be tested according to emotion databases 
in order to augment classification accuracy. A part from that, our Matlab script has 
to be cleaned up in order to increase calculation velocity and be able to send epocs 
in a time frame below 1 second. In that way, participants will be able to see a major 
influence, or at least a more continuum one, of BCI signals into the table. 

As it has been pointed out in some interviews, creating a GUI to customise mappings 
will help to reinforce a perception link between users affective states and music fea¬ 
tures. Indeed, create a link between two tactile screen GUIs, one representing valence 
and arousal in a bidimensional space and other with the music parameters that can be 
manipulated. Moreover, it can be useful to create a kind of switch button, between local 
and global, in order to get best features from each of them. 

The prime decision of only changing timbre parameters, with the manipulation of effects 
and filters, can be restructured in order to change other music parameters, such as tonal¬ 
ity, loudness and tempo, which are more directly related to music emotions according 
to [3] and [19]. In this manner, the effect of BCI over music composition will tend to be 
more clear, despite it could lead to a lose of control over the composition. 

Besides, other experimental designs that include external listener ratings on music ex¬ 
pressiveness can be proposed. As well, musicians can rate their own music compositions 
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and try to point out interesting events happening in time. Correlating afterwards brain 
log measures with those events with a software platform like Repovizz [30] , will probably 
tell us further useful information on music expressiveness. 
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